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ABSTRACT 

The denoising of video data should take into account both temporal and spatial dimensions, however, true 
3D transforms are rarely used for video denoising. Separable 3-D transforms have artifacts that degrade their 
performance in applications. This paper describes the design and application of the non-separable oriented 3-D 
dual-tree wavelet transform for video denoising. This transform gives a motion-based multi-scale decomposition 
for video — it isolates in its subbands motion along different directions. In addition, we investigate the denoising 
of video using the 2-D and 3-D dual-tree oriented wavelet transforms, where the 2-D transform is applied to each 
frame individually. 
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1. INTRODUCTION 

Although video is a 3-D data set, 3-D data transforms are usually not used for its representation — indeed, the 
standard 3-D data transforms do not provide useful representations for most video data. (By useful, we mean 
that the transform should have a good energy compaction property and that the transform coefficients reflect 
properties of the data in which we are interested.) Note that most 3-D data transforms axe separable imple- 
mentations of 1-D transforms. However, separable implementations of multidimensional (M-D) transforms have 
artifacts that can seriously reduce their usefulness, especially for the representation of M-D image data (natural 
images, video, medical volume data, etc). For example, the standard separable 3-D transform is rarely used for 
video representation because it mixes 3-D orientations in its subbands. This artifact, which is illustrated below 
and in accompanying video cHps on the web, prevents the transform from providing an effective representation 
of video. In order for a 3-D data transform to provide a useful representation for M-D image data, the transform 
probably has to be non-separable. However, non-separable implementations are more computationally expensive; 
and in addition, they are often more difficult to design. 

It turns out, that a recently developed type of complex wavelet transform can be used to implement fast non- 
separable real-valued multidimensional wavelet transforms that overcome the serious artifacts of the separable 
M-D wavelet transform. In this paper, we propose to use the 3-D dual-tree wavelet transform for video noise 
reduction. 

Although the dual-tree transform is expansive (it expands an TV-sample data vector to M wavelet coefficients 
with M > iV), it is free of the mixing artifact and gives a meaningful multi-scale decomposition for video. With 
the dual-tree transform, it is more likely that the multiresolution framework, which has proven very effective for 
image denoising can also be effectively applied to video denoising. The dual-tree transform isolates motion in 
different directions in separate subbands; the direction of motion can be inferred to some degree from the wavelet 
coefficients. Because this oriented 3-D transform can represent motion information, it provides a tool for video 
denoising that takes into account the motion of image elements, without explicitly using motion estimation. 

Programs for 3-D oriented wavelet transforms and examples will be available on the web at 
http : //taco . poly . edu/WaveletSof tware/ . 
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Figure 1. Typical wavelets associated with the 2-D separable discrete wavelet transform. The top row illustrates the 
wavelets in the spatial domain, the second row illustrates the (idealized) support of the spectrum of eadi wavelet m the 
2-D frequency plane. The checkerboard artifact of the third wavelets is apparent m the frequency domain as well as the 
spatial domain. 

2. MULTIDIMENSIONAL WAVELET TRANSFORIV^ 

In a sense, wavelet-bases are optimal for a large class of one dimensional signals (including many real signals). 
However, as has been recognized, the (separable) 2-D wavelet transform does not possess these optimality 
properties for natural images..^ The reason for this is that while the separable 2-D wavelet transform represents 
point-singularities efficiently^ it is inefficient hi line- and curve-singularities (edges)/ (Even so, it is better than 
the PCT for this — which explains the JPEG-2000 compression standard.) . . ^■ 

Some of the important developments in recent wavelet research has been the implementation of 2-D multiscale 
transforms that represent edges more efficiently than does the separable wavelet transform. Examples include 
cuiTelets,2^3 directional fflterbaiiks and pyraniids,^'^^ coniplex filter banks,^ the steerable pyramid,'^' ^ and the 
complex dual-tree wavelet transform.^' These transforms give superior results for image processing applications 
compared to the separable wavelet transform. In this paper, we investigate the use of the 3-D version of the 
dual-tree complex wavelet transform for video noise reduction. The dual-tree wavelet transform is nearly-shift 
invariant, isolates ^dges with different orientations in different subbands, and has a manageable redundancy. It 
will also be of interest to extend to 3-D, the curvelet'transform, steerable pyraniid,''and other oriented multiscale 
transforms^; and to investigate their use for video denoising. 

^ To understand why the dual-tree wavelet transform outperforms the separable wavelet transform, it is useful 
to compare the wavelets. associated with the two transforms. They axe illustrated in Figures 1 and 2 where it is 
shown that the separable 2-b wavelet transform is characterized by three wavelets, the dual-tree 2-D transform by 
six wavelets. The third wavelet associated with the separable wavelet transform has a checkerboard appearance 
— because it mixes the 45 and -45 degree, orientations; equivalently, the separable transform' fails to isolate these 
orientations. On .the other'hand, the dual-tree transform succeeds in isolating different orientations — each of 
the six wavelets are aligned along a specific direction and no checkerboard effect appears. Moreover, the dual-tree 
transform covers more distinct orientations^than does the separable transform. • 

One way to understand the checkerboard artifact that arises in the separable implementation of the 2-D 
wavelet transform is in the frequency domain. If 'ijj{x) is a real wavelet and the 2-D separable wavelet is given 
by i^{x, y) = ip{x) il){y), then the spectrum of V^(x, y) is illustrated by' the following idealized diagram. 



Figure 2. Typiced wavelets associated with the 2-D dual- tree oriented wavelet transform. The top row illustrates the 
wavelets in the spatial domain, the second row illustrates the (idealized) support of the spectrum of each wavelet in 
the 2-D frequency plane. The absence of the checkerboard phenomenon is observed in both the spatial and frequency 
domains. 



Because Tp{x) is a real function its spectrum must be two-sided; therefore, the checkerboard artifact is unavoidable 
in the frequency domain. Likewise, the checkerboard artifact arises in the spatial domain as well. 

To explain how the dual-tree complex wavelet transform produces oriented wavelets, consider the 2-D wavelet 
ip{x,y) = i^{x)ip{y) associated with the row-column implementation of the DWT, where V^(x) is a complex 
(approximately analytic) wavelet given by tlj{x) = ipfi{x) -\- jt/;g{x). We obtain for '^{x,y) the expression, 

^(a:,2/) = [V'/iW +i^ff(a:)] [(l)h{y) +3'^g{y)] (1) 
= '^h{x) if^hiy) - i^giy) + 3 [rl)g{x) tl^hiy) + ^h{x) '(pg{y)] (2) 

The support of the spectrum of this complex wavelet is illustrated by the following idealized diagram. 
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Because the spectrum of the (approximately) analytic 1-D wavelet is supported on only one side of the frequency 
axis, the spectrum of the complex 2-D wavelet ^(x, y) is supported in only one quadrant of the 2-D frequency 
plane. For this reason, the complex 2-D wavelet is oriented in a specific direction. If we take the real part of this 
complex wavelet, we obtain the sum of two separable wavelets: 



Real Part{^(x, y)} = iIjh{x) i/^hiy) - i)g{x) 'ipgiy) 



(3) 



Because the spectrum of a real function must be symmetric with respect to the origin, the spectrum of this real 
wavelet is supported in the two quadrants of the 2-D frequency plane, as illustrated in the following (idealized) 
diagram. 



Real Part { 



Unlike the real separable wavelet, the support of the spectrum of this real wavelet does not posses the checker- 
. board artifact and therefore the real wavelet is oriented. Note that this construction depends on the fcomplex 
wavelet ^p{x) = Mf'rofaix) being (approximately) analytic; or equiValently, that is approximately the 
Hilbert transform of iPh^t), [ ^g(t) « H{tPh{t)}]. . ^ • 

iifg(t) ~ nitPhit)}., then the same derivation can be used to unplement an oriented nonseparable 2-D wavelet 
transform by combining the subbands of two separable 2-D DWTs. If we define two separable 2-D wavelet bases 
in the usual manner, , . 

^iAx,y)=.4>h{x)My) i^2,i{x, y) = <Pg{x)^g{y) . . (4) 

^h2{x,y) = iPhix)My) ^i,2{x,y) = ipg{x)<t>g{y) • ■ .(5) 

i^iA^,y) = Mx)My) ^2,z{^,y) = ipg{x)iJgiy). . (6) 

then the 6 wavelets defined by . 

^*(^'f) = ;^(V'M(a;,y)+V2,<(a:,y)) " (7)- 
^i+3ix^y) = ^i'4>iAx,y)-ih,i{x,y)) (8) 

for 1 J i < 3 are oriented, as illustrated in Figure 2. Because the sum/difference operation is orthonormal, this 
constitutes a perfect reconstruction wavelet transform. (If the 1-D wavelets ^,(t) and M*) form orthonormal 
bases, then this constitutes a tight frame, or a self-inveHing transform.) • • 

In 3-D, the checkerboard artifact of the separable transform is more serious than in 2-D. Correspondingly 
dimen'^k.i''"'^'*^'^ ^'^''^^ transform in place of the separable one is greater in higher 

2.1. Developing the Fast Oriented 2-D Wavelet Ttansform 

The dual-tree wavelet transform is implemented using separable transforms and by combining subband signals 
appropriately. So even though it is non-separable (and therefore free of some of the limitations of separable 
transforms) it inherits the computational efficiency of separable transforms. Specifically, the 1-D dual-tree 
wavelet transform is implemented using two filter banks in paraUel operating on the same data a^ illustrated 
m Figure 3. A complex-valued wavelet m can be obtained as ^(t) = Mt) + i%{t) where Mt) and Mt) 
are both real-valued wavelets. To implement directional 2-D transforms using the dual-tree transform, it is 
necessary that the spectrum of the complex-valued wavelet ^(«) be single sided (^(o;) = 0 for u.< 0) It is 
therefore required that the wavelet ^|Jg{t) be the Hilbert transform of the wavelet V/,(t), ' 

• *^('"^-t j*,(a>), a;<0 ■ . • (9) 

which we denote as ^^(t) = W{Vfc(t)}. But how does one design and implement two discrete wavelet transforms so 
that the wavelets associated with them form a Hilbert pair? Specifically, how should the two lowpass CQF filters 
ho{n) and go{n) in Figure 3 satisfying perfect reconstruction properties be designed so that V<,(t) = Wibumf 
These questions have been addressed in our recent work.^^' ' 

2.2. Hilbert Pairs of Wavelet Bases: Characterization Theorem 

Hubert pairs of wavelet bases were studied in"' ^2 which give a characterization and provides a Daubechies-like 
construction for approximate Hilbert pairs of orthonormal (and biorthogonal) wavelets with vanishing moments 
and compact support. To outUne these results, let the real filters fto(n), /ii(n) represent a conjugate quadrature 
falter (CQF) pair. That is, the autocorrelation phin) of the lowpass filter ho{n) is halfband: ph(2n) = S(n) 
where pHjn) := /io(n) * fto(-n); and the highpass filter is given by /ii(n) = (-1)" ho{M - n) where M is an odd 
integer. Let the real filters 9o{n), gi{n) represent a second CQF pair. The real-valued scaling function 0^(0 and 




Figure 3. The 1-D dual-tree wavelet transform is implemented using a pair of filter banks operating on the same data 

simultaneously. The upper iterated filter hank represents the real part of a complex wavelet tr2msform. The lower one 
represents the imaginary part. The transform is an expansive (or oversampled) trsinsform (or frame). 



the real- valued wavelet iph{t) associated with the pair {hQ{n), hi{n)) axe defined implicitly by the following pair 
of equations: 

Mt) = 5] hoin) ij>H{2t ^ n), ^/,(t) = >/2 J]) hi{n) (t>h{2t - n). (10) 

n n 

The real- valued scaling function (l)g{t) and wavelet ipg{t) eissGciated with the pair {9o{n),gi{n)) are defined 

similarly. Note that if |fro(e^**')| = \Gfi{e^'^)\ then it can be shown that = Therefore, the two 
lowpass filters should be related as 

Goiuj)^Ho{uj)e-^^^''^ (11) 

where 0{uj) is 27r-periodic. But how should the phase function 6{uj) be chosen so that the two wavelets generated 
by ho{n) and ^o(^) form a Hilbert transform pair? Using the infinite-product formula it can be shown that (11) 
leads to the following relationship between the wavelet spectrums: 



^g{uj) = ^H{(^) exp j 



k=2 

Prom (9) and (12) we see that 9{iu) must satisfy the condition: 



(12) 



9{u/2-n)-f^0iuj/2')^{ I (13) 

We have shown in^^ that if 6{u)) is given by 6{ijj) = 0.5a; for |a;| < 7r then condition (13) holds and ijg{t) = 

Theorem: If Hoie^"^) and Go{e^'^) are lowpass CQF filters with 

Go(e^*'") = i?o(e^")e-^^-^" for M < tt, (14) 
then the corresponding wavelets are a Hilbert transform pair, i)g{t) = H{i)h{t)}, 

Equivalently, the digitgJ filter go{n) is a half-sample delayed version of ho{n), go{n) = /io(n-0.5). If it is desired 
that both lowpass filters ho{n) and po(^) be FIR, then it is impossible to satisfy (14) exactly and an approximation 
must be made. Informally, if in the time-domain we have go{n) w ho{n — 0.5), then ipg{t) ~ 'H{iph{t)}. In^^ 
we have developed a Daubechies-like algorithm for the construction of Hilbert pairs of short orthonormal (and 
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^i^rAtTZvl h W^^^ T ° wavelet ba^es designed according to.^^ Left: The wavelets 

fh(t), are shown by blue and green sohd hnes. The magnitude of the complex wavelet Mt) = i^M+jM) is 

rr^^leS'EhL^iS^r^^^^^ wavelet tra^form axe efficiently implemented using :HUb«, pair, W 

Worthogonal) wavelet bas^^ 

• w^lr f r' . . ' "^'^ *° implement directional 2-D wavelet transforms, the associated 

wavelets of which axe illustrated m Figure 2. In addition, these pairs of filters can be used to efficiently implement 
the motion-selective wavelet transform. ■> .. if , "="" 

3. MOTION-BASED WAVELET TRANSFORM FOR VTOEO REPRESENTATION 

The shortcoming of the separable 2-D wavelet transform for image processing is further compounded for video 
mTTw~r ^ u?""^"""? "^P"^^^^^ transforms even more mixing- of different orientations occurs." 

(Note that with video, an object edge moving in a particular- spatial orientation contributes to a 3-D edge ) 
2l!^J'T' u^g^'^'^'^f transforms have not been widely appUed in video restoration. Previous works on 
^deo restoration ^ usuaQy perform noise reduction along motion trajectories. In some algorithms, a separable 
wavekt transform is apphed along themotion trajectory in. the temporal direction: i.e. first determine the motion 
trajectory of a pixel or block, then process along that trajectory in the temporal direction. The performance 
however, is dependent on the accuracy of motion estimation which is more difficult for noisy video. 

The principle by which the 2-D dual-tree DWT resolves the problem of the mixing of orientations, can also be 
w^t tV T^""^ of orientations in the 3-D case. This dual-tree wavelet transform is moHor^selective, 

Zht It f P^*^^% 3-D wavelet transform is not., (Likewise, the 2-D dual,tree transform is. direcUon-sekctivel 
while the 2-D transform is not.) • ■ " 

, Figures 5 and 6 iUu^rates the difference between the separable 3-D wavelet .transform and the dual-tree 3-D 
transform. The figm-es depict a typical 3-D wavelet V(x, y, t) associated with each of the two wavelet transforms 
The isosurface provides a way to represent a function of three variables, similar to a contour plot of a function 
of two variables To interpret the isosurfaces in Figure 5 note^that, like a contour , plot, the points on the 
curves (surfaces) are points where the functions are equal-valued. The wavelet associated with the separvhle S-D 
transform has the same checkerboard phenomenon present in the sepamble 2-D case, a consequence of the mixing 
of orientations^ The wavelet associated with the dual-t,^ S-D tmnsform is free of this effect, as iUustrated in 
fSh K, r/'^^ °^ 1"^^ '^P'- 3-D transform has many more subbands 

SJ^tL^^idtiin^^^^^ 

b..tw'tr "^"v accompany this paper (available on the web) reveal the behavior o'f these 3-D functions 

better than Figure 6 can. When the wavelets of the dual-tree 3-D transform are displayed as' a hiovie," one 
obsenres a fimte-time motion in a localized patch in a particular direction - the dual-tree 3-D wavelet transform - 
therefore reveals motion and edge information. The subbands of the duai:tree 3-D wavelet transform can isolate 
2-D ,edges with different orientations that are moving in differait directions; The video' 'clips :show,' oh the'other.- 



hand, that the separable wavelet conveys/captures limited motion information and conveys/captures limited 
edge information in a video frame. 

The idea of using a complex (analytic) wavelet transform to implement a real 3-D oriented wavelet transform 
for motion analysis was described originally by Burns et a/.^°' The implementation described in these references 
also uses the Hilbert transform to avoid the checkerboard artifact. In these papers, the Hilbert transform is 
first applied to the data, the separable wavelet transform is applied to both the original data and the Hilbert 
transformed data, and then the wavelet coefficients are combined as in the dual- tree transform, to obtain an 
oriented transform. On the other hand, in the dual-tree implementation, the Hilbert transform is built into the 
wavelet transform itself. 

Note that the ideal Hilbert transform is represented by an infinitely long impulse response that has slow 
decay. The use of the ideal (or near ideal) Hilbert transform in conjunction with the wavelet transform effectively 
increases the support of the wavelets. In order that the wavelets have short support, an approximate Hilbert 
transform that is more locaHzed in time should be used instead. However, the accuracy of the approximate 
Hilbert transform should depend on the scale of the wavelet transform. When the Hilbert transform is applied 
first to the data, a single Hilbert transform is effecting wavelet coefficients at all scales, while when the Hilbert 
transform is built into wavelet transform as in the dual-tree implementation, the Hilbert transform scales with 
the wavelet coefficients. 

The 3-D oriented dual-tree wavelet transform is developed similarly to the 2-D version. Suppose that ipgit) 
is approximately the Hilbert transform of '0/i(O> bPgW ^ ^{i^hi^)}]' Define the approximately analytic complex 
wavelet ^(•) by ^(x) = ^/i(x) -f- jf ^^(x). Then consider the complex 3-D wavelet, 

i^ix,'^]z)='ilj{x)'tp{y)rl;{z) (15) 

% = (V';,(x) + j v^,(x)) {My)-^jMy)) (16) 

As in the 2-D case, take the real part of -^(x, y, z) to get an oriented real 3-D wavelet: 

Real FaTt{iP{x, y, z)} = rPh{x) iph{y) iph{z) - ipg{x) iPg{y) ij^{z) - i;g{x) tPniy) i^g{z) - ^h(x) ipg{y) il)g{z). 
We can write it more compactly as 

Real Part{V'(x, y, z)} = rpi{x, y, z) - -02 (x, y, z) - ip^{x, y, z) - -04 (x, t/, z) 

where we define 



^i(x, 2/, z) := iphix) i)h{y) tph{z) (17) 

V'2(x, y, z) := 'tpg{x) i^giy) iPh(z) (18) 

tpsix^y.z) := i)g{x)i)h{y)'^g{z) (19) 

^4(x, y, z) := 'iphix) ipg{y) ^tpgiz), (20) 



In the 3-D case, it is necessary to combine four separable wavelet transforms, instead of two as in the 2-D case. 
To obtai n the rema ining subbands, we take in addition to (15), the real part of 'ip{x)tp{y)ip{z), 'ip{x)rp{y)il){z)^ 
and il^{x)il^{y)7lj{z)y where the overfine represents complex conjugation. This gives the following orthonormal 
combination matrix of the four separable 3-D wavelet transforms: 

^a(x, y, z) = 0.5 (^1 (x, y, z) - -02 (x, y, z) - V'3(x, y, z) - -04 (x, y, z)) 

'^h{x,y,z) = 0.5 {il)i{x,y,z) - ^2(x,y,2:) + V'3(x,y,2) -h^4(x,y,z)) 
ipcix, y, z) = 0.5 (^1 (x, y, z) -h ^2(x, y, z) - Tpsix, y, z) + ip^ (x, y , z)) 
^d(x, y, z) = 0.5 (^1 (x, y, z) -h ip2{x, y, z) 4- ip^{x, y, z) - V'4(x, y, z)) 

By applying this combination matrix to each of the seven subbands, one obtains the 3-D oriented dual-tree 
wavelet transform illustrated in Figure 6. A program implementing this oriented transform is available on the 
web. 




■ Figure 5. Isosurfaces of the 7 3^D wavelets for a typical separable 3-D wavelet transform. The blue and red surfaces 
represent the points where the wavelet ip{x,y,t) is equal- valued (blue: 'tp{x,y,t) = 0.3; red: 'ipix.y.t) = -0.3).. 




Figure 6. Isosurfaces of 7 of the 28 3-D wavelets for a typicaJ oriented 3-D dual-tree wavelet transform. For the dual-tree 
3-D wavelet transform, each subband corresponds to motion in a specific direction. Video clips available on the web 
display the wavelets as evolving gray-scale images. 



Figure 7. Denoising results for frame 36 of the test sequence. ■ r ' ; i 
4. DENOISING EXAMPLE 

aad .he 3-D orlentoj w»^et .r^Sorrtol to d^^^^ XteTu^'J '"^^'^ 
fa.t motion it is more difficult'to explpit the tlptd dltt^^^^^ 
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Original (Frame 21) 2D Separable (Frame 21) 3D Separable (Frame 21) 




Noisy (Frame 21 ) 2D Oriented (Frame 21 ) 3D Oriented (Frame 21 ) 




Figure 8. Denoising results for frame 21 of the test sequence. 
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